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Executive  Summary 

This  document  is  the  final  report  on  research  carried  out  by  QinetiQ,  Centre  for 
Human  Sciences  on  behalf  of  the  European  Office  of  Aerospace  Research  and 
Development  under  contract  No.  FA8655-03-1-3048. 

The  research  was  aimed  at  a  preliminary  investigation  into  whether  the  provision  of 
vibrotactile  signals  might  benefit  pilots  who  were  experiencing  spatial  disorientation. 

The  objectives  were  to:  - 


•  Determine  experimentally  whether,  and  by  how  much,  a  tactile  signal  can 
benefit  an  individual  carrying  out  a  high  workload  visual  task. 

•  Determine  and  quantify  the  effects  of  congruent  and  incongruent  tactile- 
visual  information  on  performance. 

•  Ascertain  if  any  beneficial  effect  is  as  a  result  of  true  multisensory 
integration  or  whether  it  is  a  simple  alerting  (attention  grabbing)  effect. 

We  have  used  a  simulation  of  a  pilot  activity  in  order  to  investigate  these  issues. 
Subjects  carried  out  a  visual  alphabet  arithmetic  task  along  with  a  simultaneous 
visual  tracking  task.  The  tracking  task  was  completed  with  no  tactile  cueing,  or 
augmented  incongruent  or  congruent  tactile  cueing.  The  results  demonstrated  that 
providing  subjects  with  a  tactile  stimulus  whilst  undertaking  visual  tasks  does  not 
distract  a  subject  from  these  main  visual  tasks.  This  is  evidenced  by  the  maintained 
performance  in  the  alphabet  calculation  task,  across  all  of  the  experimental 
conditions. 

Tactile  cueing  did  provide  a  benefit  in  terms  of  response  time  in  reacting  to  the 
tracking  task.  Response  times  with  tactor  support  were  approximately  twice  as  fast 
compared  with  no  tactor  support.  In  addition,  there  was  a  much-reduced  variability 
in  response  times  with  tactor  support  compared  with  none. 

Small  differences  in  congruent  and  incongruent  tactor  cueing  were  observed  in  the 
data  from  the  first  experiment.  The  second  experiment  demonstrated  that  the  main 
effect  of  the  tactor  was  to  act  as  a  cue  to  the  track  being  lost,  i.  e.  as  an  attention 
grabbing  mechanism.  There  was  no  direct  benefit  due  to  the  lateralisation  of  the  cue. 

Overall  the  results  showed  that  subjects  were  able  to  increase  the  number  of  tracking 
corrections  without  any  detriment  to  the  high  demand  cognitive  task,  when  tactor 
cueing  was  used.  One  might  therefore  infer  an  increased  capacity  as  a  result.  The 
results  support  the  concept  of  using  a  tactor  system  as  a  method  of  aiding  a  pilot  by 
reducing  workload  and  improving  performance  by  producing  faster  and  more 
consistent  reaction  times.  Tentatively,  the  results  suggest  that  tactile  systems  may 
reduce  the  risk  of  pilot  spatial  disorientation  by  two  methods.  Firstly,  by  reducing 
primary  task  load,  enabling  improved  performance  on  a  secondary  task  (e.g.  target 
acquisition).  Secondly,  by  providing  an  attention-grabbing  mechanism  for  pilots  who 
have  become  spatially  disorientated  and  are  unaware  of  the  fact  (Type  I  SD). 
However,  there  are  still  many  human  factors  issues  regarding  the  integration  of 
tactile  systems  for  pilots  that  require  further,  systematic  investigation. 

We  recommend  that  further  work  should  be  undertaken  to  investigate  the  benefits 
identified  in  this  study.  A  suitable  environment  in  which  to  undertake  such  studies 
would  employ  immersive  simulation  to  increase  the  ecological  validity  of  the  task, 
whilst  maintaining  experimental  control.  An  ideal  test-bed  for  this  kind  of  approach 
would  be  a  Cognitive  Cockpit  environment. 
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Introduction 


Background 

This  document  is  the  final  report  on  research  carried  out  by  QinetiQ,  Centre  for 
Human  Sciences  on  behalf  of  the  European  Office  of  Aerospace  Research  and 
Development  under  contract  No.  FA8655-03-1-3048. 

Spatial  disorientation  is  a  major  cause  of  both  military  and  civilian  aircraft  accidents 
[1].  It  was  a  major  contributory  factor  in  30%  of  all  US  Army  rotary-wing  accidents 
which  occurred  between  1987  and  1995,  and  which  were  graded  as  class  A1. 
Accidents  arising  from  spatial  disorientation  were  more  severe  than  those  not 
involving  spatial  disorientation,  with  36%  of  accidents  in  Class  A  compared  to  18%  of 
non-disorientation  accidents.  In  the  10-year  period  between  1991-2000  inclusive, 
spatial  disorientation  was  the  cause,  or  a  major  contributory  factor,  in  20%  of  USAF 
accidents,  at  the  cost  of  60  aircrew  lives  and  $1.4B. 

Clearly,  there  is  value  in  both  humanitarian  and  financial  terms  of  investigating 
methods  that  reduce  the  likelihood  of  spatial  disorientation.  At  present,  all 
orientation  information  is  presented  to  the  pilot  visually.  This  modality  is  particularly 
susceptible  to  becoming  overloaded,  leading  to  a  loss  of  [correct]  spatial  orientation. 

Since  humans  have  evolved  to  integrate  information  from  multiple  senses,  it  is 
feasible  that  by  providing  additional  information  through  other  senses,  to  support 
decisions  on  orientation,  there  might  me  a  way  of  reducing  loss  of  correct  spatial 
orientation.  This  report  presents  the  results  of  a  preliminary  study,  which 
investigated  whether  using  a  combination  of  visual  and  tactile  information  would 
improve  pilot  performance  and  reduce  the  risk  of  spatial  disorientation. 

Spatial  Disorientation  in  Aircraft 

Accurate  perception  of  orientation  within  the  external  world  is  achieved  by  the 
combined  use  of  visual,  vestibular  and  somatosensory  information  [2].  This  sensory 
information  is  compared  to  previous  experience,  and  a  mental  model  of  the 
orientation  and  position  of  the  individual  in  the  external  world  is  formed  (see  Figure 
1-1).  Vision  is  the  key  sense  for  orientation,  and  normally  acts  as  a  frame  of  reference 
for  the  interpretation  of  information  from  the  non-visual  senses. 


Vestibular  Somatosensory 


Visual 


Integration  and  interpretation  in 
terms  of  past  and  current  experience 


Mental  model  of 
spatial  orientation 


1  A  Class  A  accident  is  defined  to  be  an  accident  that  results  in  fatality  or  total  permanent 
disability,  loss  of  an  aircraft,  or  property  damage  of  $1  million  or  more. 
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Figure  1-1  Representation  of  the  sensory  inputs  and  processes  for  spatial  orientation 


Aircraft,  both  rotary  and  fixed-wing,  can  create  dynamic  environments  in  which  it  is 
difficult  for  the  pilot  to  maintain  the  correct  spatial  orientation,  or  where  it  is  difficult 
for  the  pilot  to  regain  the  correct  orientation  when  it  is  lost  [3].  Without  adequate 
visual  cues,  a  condition  that  can  occur  when  flying,  during  instrument  meteorological 
conditions  (IMC)  or  at  night,  the  dynamic  environment  of  the  aircraft  may  induce  an 
erroneous  perception  of  orientation.  For  example,  a  helicopter  might  drift  or  rotate 
during  a  hover  manoeuvre,  but  the  movement  may  be  below  the  detection  threshold 
of  the  vestibular  system  of  the  pilot.  Without  the  concomitant  visual  motion  cues, 
the  sub-threshold  linear  drift  or  rotation  of  the  helicopter  may  lead  the  pilot  to 
believe  he  or  she  is  maintaining  the  same  position  and  heading,  when  in  fact  the 
aircraft  may  have  side  slipped  or  be  facing  in  a  different  direction. 

Inadequate  or  erroneous  visual  cues  alone  can  also  cause  spatial  disorientation.  For 
example,  judging  height  can  be  difficult  when  flying  over  featureless  terrain  such  as  a 
sandy  desert,  due  to  the  lack  of  texture  cues  [4].  The  assumption  that  cloud  tops  are 
always  horizontal  may  cause  disorientation  when  they  are  not  actually  horizontal  [5], 
In  this  situation,  the  dynamics  of  the  flight  manoeuvres  give  rise  to  accelerations 
which,  when  combined  with  gravity,  feel  like  gravity  alone  to  the  pilot,  but  are  no 
longer  indicative  of  the  true  vertical.  Such  sensations  also  occur  in  turns  and 
manoeuvres  that  involve  deceleration  or  acceleration  in  the  line  of  flight  [6]. 

Spatial  disorientation  can  also  occur  if  the  pilot  has  been  engaged  in  a  secondary  task, 
where  distractions  could  cause  the  pilot  to  miss  vital  flight  data  from  instruments  [7], 

When  flying  in  conditions  of  poor  visibility,  such  as  in  cloud  or  at  night,  the  only  visual 
orientation  reference  may  be  the  environment  of  the  cockpit.  In  these  situations 
pilots  become  heavily  reliant  on  information  from  instrument  panels,  night  vision 
devices,  and  from  helmet-mounted  and  head-up  displays,  to  maintain  spatial 
orientation.  Any  kind  of  distraction  from  these  tasks  may  result  in  missed  changes  in 
aircraft  orientation  information.  Figure  1-2  illustrates,  within  the  cognitive 
framework  of  figure  1-1,  how  information  from  flight  instruments  is  utilised  for 
determining  pilot  spatial  orientation. 


Visual 


Vestibular 


Somatosensory 


4 . / 


4 


Scanning  of 
flight  data  and 
interpretation 


Mental  model  of 
spatial  orientation 

Figure  1-2  Illustration  of  how  sensory  information  and  data  gained  from  flight 
instruments  determine  pilot  spatial  orientation 
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Although  most  military  aircrew  receive  some  form  of  spatial  disorientation  training 
(lectures  and/or  ground-based  demonstrations,  and/or  in-flight  demonstrations)  the 
rate  of  aircraft  accidents  attributable  to  SD  does  not  appear  to  be  diminishing.  For 
example,  SD  has  been  cited  as  a  main  causal  factor  in  4-5%  of  US  military  aircraft 
accidents  (Army,  Navy  and  Air  Force)  over  the  last  5  -10  years  [8], 

Conveying  aircraft  orientation  cues  by  a  more  intuitive  method  than  the  current 
interpretation  of  visual  displays  may  be  of  benefit  in  reducing  spatial  disorientation 
by  augmenting  sensory  information.  Intuitive  cues  may  not  only  enable  a  rapid 
recovery  of  the  aircraft  from  an  unusual  attitude  but  may  also  reduce  the  likelihood 
of  this  situation  arising  in  the  first  place,  by  improving  the  ease  by  which  pilots 
maintain  their  mental  model  of  aircraft  orientation.  This  could  be  achieved  by  either 
using  these  cues  to  reduce  visual  and  mental  workload  on  secondary  tasks,  therefore 
freeing  resources  for  frequent  and  adequate  scanning  of  instruments,  or  as  an 
orientation  aid  per  se. 

Research  into  alterative  means  of  conveying  aircraft  orientation  information  has 
been  underway  for  some  years.  These  include  the  use  of  the  auditory  modality,  such 
as  3D  audio  cues  [9]  and  more  recently,  the  somatosensory  sense  via  the  use  of  tactile 
devices  [10].  For  the  purposes  of  this  study  and  report,  the  latter  will  be  further 
described  and  examined. 

Tactile  aids  to  flying,  such  as  the  Tactile  Situation  Awareness  System  (TSAS), 
developed  by  the  Naval  Aeromedical  Research  Laboratory,  have  been  shown  to 
effectively  aid  the  pilot's  maintenance  of  situation  awareness  and  the  correct  spatial 
orientation  [11,  12, 13].  Tactile  aids  to  flying  offer  the  possibility  of  reducing  the  risk 
of  developing  SD  in-flight  in  the  following  three  ways.  Firstly,  tactile  information 
could  be  highly  intuitive,  and  therefore  require  less  cognitive  processing  or  could  be 
learned  to  a  level  where  their  processing  is  automatic.  As  such,  they  might  therefore 
help  reduce  visual  workload  during  high  workload  situations  such  as  that 
encountered  during  air-to-air  combat  by  moving  the  task  into  a  separate  sensory 
modality.  Secondly,  tactile  aids  have  been  shown  to  vastly  improve  the  detection  of 
changes  in  aircraft  attitude  that  occur  at  when  the  movements  are  insufficient  to 
stimulate  the  vestibular  system  (passive  somatogravic  illusions,  e.g.  undetected  drift 
of  rotary  craft).  This  benefit  could  also  transfer  to  fixed-wing  aircraft  reducing  the 
incidence  of  both  passive  and  active  somatogravic  illusions  (the  latter  are  generated 
by  supra-threshold  vestibular  stimulation,  such  as  acceleration  in  fixed-wing  aircraft 
that  could  otherwise  lead  to  a  false  sense  of  pitch  up).  Finally,  tactile  aids  have  been 
shown  to  enable  pilots  to  recover  from  unusual  attitudes  to  straight  and  level  flight 
in  the  absence  of  flight  instruments  [11],  Overall,  tactile  aids  that  relay  spatial 
orientation  information  may  reduce  the  risk  of  unrecognised,  Type  I  SD  (first  and 
second  points  above)  and  improve  recovery  from  recognized,  Type  II  SD  (third  point). 

While  demonstrations  and  trials  of  tactile  aids  in-flight  and  in  simulator 
environments  have  illustrated  their  effectiveness  at  reducing  the  risk  of  SD, 
fundamental  human  factors  research  to  quantify  and  understand  the  mechanisms 
that  might  give  rise  to  benefits  has  not  been  fully  conducted.  Existing  and  previous 
research  programmes,  centre  on  implementation  details  such  as  tactor  placement, 
number  of  factors,  frequency  of  tactor  vibration,  and  duration  of  vibration  for  the 
optimum  relay  of  tactile  information  in  stationary  situations  [14],  A  number  of 
fundamental  questions  about  multisensory  integration  still  require  investigation. 
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This  report  summarises  preliminary  experiments  to  examine  the  following: 

•  How  is  sensory  information  attenuated  in  high-workload  situations? 

•  If  the  information  is  congruent2,  does  this  result  in  improvements  in 
performance? 

•  If  the  information  is  incongruent,  does  this  increase  workload  and  result 
in  decrements  in  performance? 

•  In  a  multi-sensory  situation,  which  sense  dominates? 


The  remainder  of  the  report  is  organised  as  follows: 

Section  2  provides  a  short  review  of  current  understanding  of  multisensory 
integration  from  a  neurological  and  psychological  perspective  along  with  a  review  of 
recent  related  work  on  tactors.  Section  3  provides  a  description  of  the  scoping 
experimental  programme  we  have  undertaken  in  an  attempt  to  address  some  of  the 
above  questions,  and  section  4  provides  a  summary  of  the  conclusions  and 
recommendations  for  future  work.  Appendix  A  comprises  a  summary  of  a  literature 
review  carried  out  on  tactor  threshold  studies. 


2  Throughout  the  report  we  will  use  the  words  congruent  to  signify  inputs  to  the 
somatosensory  system  that  are  on  the  same  lateral  aspect  as  the  visual  information,  and 
incongruent  to  referto  inputs  that  are  on  the  opposite  side. 
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2  Multisensory  Integration 


2.1  General 

Multisensory  integration  is  one  of  the  many  tantalising  capabilities  of  the  human 
(and  presumably  animal)  brain.  Many  ordinary  events  in  the  physical  world  -  the 
scent  from  a  vase  of  flowers,  the  sound  of  a  car  passing  by,  the  feeling  of  a 
handshake,  the  sight  of  a  tree  blowing  in  the  wind  -  produce  signals  in  several 
sensory  modalities  simultaneously.  Although  much  information  about  such  events 
can  be  extracted  by  treating  these  signals  one  modality  at  a  time,  it  is  clear  that  the 
brain,  by  integrating  across  sensory  modalities,  achieves  a  sum,  which  is  greater  than 
the  parts.  Our  brain  is  exquisitely  attuned  to  multisensory  correlations,  which  it  uses 
to  modulate  and  refine  perceptual  analysis. 

There  are  numerous  examples  of  how  perception  can  be  enhanced,  but  also  biased, 
by  the  integration  of  sensory  signals  across  modalities.  For  example: 

Low  contrast  or  noisy  visual  and  sound  stimuli  can  be  combined  to  improve 
spatial  localisation.  On  the  other  hand  a  salient  visual  stimulus  can  readily 
“capture”  the  perceived  origin  of  a  sound,  as  occurs  when  we  view  a 
ventriloquist  [15]. 

Being  able  to  see  ones  arm  improves  the  spatial  resolution  of  tactile 
discrimination  when  two  (unseen)  pinpoint  stimuli  are  applied  near  one 
another  on  the  skin  [16]. 

These,  and  many  more  examples  underscore  the  importance  of  multisensory 
integration  in  assisting  perceptual  decisions. 

2.2  Neurobiology  of  Multisensory  Integration  and  Sensory  Maps 

Classically,  cross-modality  effects  have  been  attributed  to  neuronal  interactions  in 
multisensory  convergence  areas  of  the  brain  such  as  the  superior  colliculus  [16]  orthe 
parietal  lobe  [17].  However,  more  recently,  with  the  advent  of  functional  magnetic 
resonance  imaging  (fMRI)  this  traditional  view  has  been  questioned.  Multisensory 
effects  have  been  isolated  in  auditory  [18]  and  visual  [19]  cortices.  These  lines  of 
research  are  new,  and  it  is  not  yet  clear  what  the  mechanism  is  by  which  a  tactile 
stimulus  is  able  to  exert  a  modulation  on  visually  evoked  activity  in  the  occipital 
cortex.  There  appears  to  be  a  general  design  principle  within  the  brain  by  which  not 
only  are  their  forward  projections  starting  at  the  sensory  inputs,  but  there  are  also 
back-projections  from  higher  areas  towards  the  sensory  inputs  [20], 

An  additional  challenge  for  understanding  how  multisensory  integration  occurs  is  to 
determine  what  neural  computations  are  required  and  in  particular  how  the  different 
“maps”  for  the  different  senses  are  combined.  Visual,  auditory  and  somatosensory 
stimuli  are  represented  by  different  topographic  arrangements  in  their  in  their 
respective  cortical  areas.  This  is  primarily  because  each  sensory  map  is  organised 
according  to  the  geometry  of  the  corresponding  receptors.  The  bite  of  a  mosquito  on 
ones  hand  is  represented  in  the  somatosensory  cortex  in  a  skin-based  frame  of 
reference,  whereas  our  visual  image  of  the  mosquito  biting  is  coded  in  a  retinal  frame 
of  reference.  Not  only  are  these  maps  different  geometries  but  each  is  also  distorted 
according  to  the  relative  importance  (as  measured  by  cortical  area)  of  different  parts 
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of  the  sensory  surface.  Thus  in  the  visual  map,  considerable  brain  area  is  allocated  to 
processing  the  central  (foveal)  region.  The  relative  representations  of  different  parts 
of  the  body  are  shown  in  figure  2-1  and  the  proportion  of  cortex  devoted  to  different 


Figure  2-1  Diagrammatic  representation  of  the  somatosensory  mapping  to  the  cortex. 
Diagram  represents  a  sagittal  section  through  the  left  and  right  cortex  through  the 
region  representing  somatosensory  inputs.  The  sizes  of  the  different  pieces  of  the  body 
are  scaled  to  represent  their  relative  importance(6). 
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Figure  2-2  Relative  representation  of  the  visual  field  in  the  cortex(6). 
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A  minimal  requirement  for  visuo-tactile  integration  is  that  the  signals  arising  from 
the  two  separate  modalities  be  assigned  a  common  spatial  origin.  This  implies  that 
the  signals  generated  in  one  modality  need  to  be  recoded  in  the  frame  of  reference  of 
the  other  modality,  or  alternatively  that  each  modality  is  recoded  into  a  single  and 
separate  abstract  frame  of  reference.  How  this  is  achieved  is  still  an  area  of  active 
research  and  not  understood. 

Furthermore,  the  whole  integration  process  is  perforce  dynamic.  The  retinal  location 
of  a  tactile  stimulus  will  shift  whenever  the  hand  is  moved  or  a  change  of  gaze  occurs, 
therefore  the  mapping  of  spatial  coordinates  must  be  continuously  updated.  There  is 
neurophysiological  evidence  for  transformations  of  coordinates  on  the  parietal  and 
premotor  multimodal  areas  [21],  22]  as  well  as  computational  theories  [23]  as  to  how 
these  might  operate. 

Psychological  Models 

A  number  of  psychological  models  exist  to  explain  experimental  studies  into 
attention  allocation  within  a  multisensory  environment.  Flexible  limited  capacity 
models  propose  attentional  resources  are  allocated  through  a  central  pool.  Multiple 
resource  theories  on  the  other  hand  propose  that  there  are  several  resources  and 
within  each  of  these  we  can  allocate  attention  separately  [24,  25],  Resources 
represent  the  mental  effort  supplied  to  improve  the  processing  efficiency  or  the 
performance  of  a  mental  operation.  Their  deployment  is  under  voluntary  control  and 
they  are  scarce  [26], 


Tasks,  to  be  performed  concurrently,  will  interfere  more  if  more  resources  are  shared. 
Wicken’s  model  [25]  tries  to  account  for  the  allocation  of  attention  and  the 
processing  of  information  in  complex  tasks,  such  as  piloting  an  aircraft,  in  which  the 
operator  works  in  a  high-workload  environment  and  has  difficulty  processing  all  the 
information.  Figure  2-3  presents  this  model,  in  which  resources  are  defined  by  three 
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relatively  simple  dichotomous  dimensions:  two  resources  defined  by  processing  stage 
(early  versus  late  processes),  two  modality-defined  resources  (auditory  versus  visual 
encoding),  and  two  resources  defined  by  processing  codes  (spatial  versus  verbal). 

In  the  stage  dimension,  the  resources  used  for  perceptual  and  central-processing 
activities  appear  to  be  the  same,  and  they  are  functionally  separate  from  those 
underlying  the  selection  and  execution  of  responses.  The  two  perceptual  modalities 
are  distinguished  to  account  for  the  possibility  to  divide  attention  between  two 
senses  such  as  the  eye  and  the  ear  for  example,  which  is  easier  than  between  two 
auditory  channels  or  two  visual  channels.  The  degree  to  which  peripheral  rather  than 
central  factors  are  responsible  for  better  cross-modal  time-sharing  than  intra-modal 
is  uncertain.  The  code-related  resource  dimension  may  be  defined,  in  part,  by  the  two 
cerebral  hemispheres:  spatial  processing  taking  place  mainly  in  the  right  hemisphere 
and  verbal  processing  mainly  in  the  left  hemisphere  [26],  With  respect  to  the  codes 
for  information  processing,  two  principles  of  stimulus— central-processing— 
response-compatibility  are  conveyed.  Verbal  tasks  seem  to  be  best  served  by  auditory 
inputs  and  speech  responses,  whereas  spatial  tasks  seem  to  be  best  served  by  visual 
inputs  and  manual  responses  [25,  27], 


The  multiple-resource  theory  of  Wickens  provides  a  good  framework  that 
summarises  a  lot  of  experimental  data  on  dual-task  performance.  However,  it  does 
not  encompass  all  possible  effects  in  dual  task  performances,  such  as  switching, 
confusion  and  cooperation  [28],  The  interplay  between  cognitive  processes  proves  to 
be  rather  complex  and  the  “simple”  attribution  of  cognitive  tasks  to  a  small  number 
of  resources  addresses  this  interplay  only  partially. 

Related  Tactor  Work 

Young  et  al  [29]  investigated  the  effect  of  cross-modal  links  in  attention  between 
haptics  and  vision  using  a  2  x  2  tactile  display  in  a  seat  back,  to  cue  a  4-quadrant 
visual  search  task.  In  this  single  task  design,  subjects  were  instructed  to  identify  the 
target  quadrant  as  soon  as  they  noticed  it,  either  with  or  without  the  aid  of  tactile 
cues.  Furthermore,  the  experimenters  employed  two  conditions  of  tactile 
stimulation,  one  in  which  there  was  80%  agreement  of  the  tactile  cues  with  the  visual 
representation  and  another  in  which  there  was  only  20%.  Subject  performance  on 
the  task  was  measured  by  recording  the  reaction  time  to  correctly  identify  the  target 
quadrant. 

The  experimenters  found  that,  in  the  80%  agreement  condition,  the  subjects  were 
able  to  consistently  improve  reaction  time  performance  over  the  no  tactile  cues 
baseline.  Performance  was  more  variable  in  the  20%  agreement  condition.  Young  et 
al  noted  that  subjects  exhibited  positive,  negative  or  no  haptic  cueing  effects.  From 
these  results  the  authors  concluded  that  vibrotactile  cues  to  orient  visuo-spatial 
attention  are  intuitive  but  not  involuntary. 

Lindeman  et  al  [30]  investigated  the  effects  of  adding  vibrotactile  information  to 
performance  on  a  visual  search  task.  In  a  similar  apparatus  to  experiment  above,  the 
experimenters  utilised  a  3x3  array  of  factors  in  a  seat  back  to  deliver  tactile 
information  to  the  subject.  Subjects  were  required  to  locate  a  target  letter  on  a 
screen  of  three  blocks  of  8  randomly  organised  distractor  letters.  The  conditions 
encompassed  a  variety  of  different  visual  cueing  conditions,  with  or  without  tactile 
aids,  and  then  a  baseline  condition  in  which  subjects  performed  the  letter  search  task 
without  any  aids.  Performance  was  measured  by  the  reaction  time  to  correctly 
identify  the  target  letter. 
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The  experimenters  found  that  visual  cueing  was  the  most  effective  performance 
enhancing  measure,  although  tactile  cueing  also  significantly  improved  performance 
over  the  baseline.  However  it  was  noted  in  the  discussion  that  the  tactors  used  in 
this  trial  were  pager  motors.  These  have  a  slight  time  delay  between  activation  and 
reaching  the  acceleration  levels  required  to  take  the  stimulus  supra-threshold. 
Compared  to  the  virtually  instantaneous  response  to  visual  cueing,  this  time  delay 
might  have  been  expected  to  affect  the  results  in  the  tactile  cueing  conditions. 

Summary 

In  summary,  the  literature  is  clear  that  congruent  tactile  stimuli  can  improve 
performance  on  search  tasks.  However,  the  results  from  these  experiments,  whilst 
useful,  are  limited  in  the  extent  to  which  they  can  be  applied  to  the  some  of  the 
proposed  real  world  applications  of  factor  technology  in  high  workload  situations. 
They  typically  involve  trials  with  one  task,  from  which  the  subject  is  not  expected  to 
divert  his/her  attention  at  all. 

Furthermore  it  is  clear  that  multisensory  integration  can  be  viewed  and  interpreted 
as  a  "low-level”  sensory  process  or  a  “high-level”  cognitive  process. 

Overall,  this  would  suggest  that  there  is  a  need  for  further  research  into  visuo-tactile 
interactions  and  integration,  with  emphasis  on  simulations  of  high  workload  tasks. 
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3  Experiment  One 

3.1  Introduction 

3.1.1  Aim 

The  aim  of  this  preliminary  experimental  program  was  to  conduct  trials  to  ascertain 
the  influence  of  congruent  and  incongruent  tactile-visual  information  on 
performance  during  an  abstraction  of  a  real  cockpit  environment 

The  results  of  this  scoping  study  could  be  used  to  direct  a  future  program  to 
systematically  investigate  the  multi-sensory  integration  and  quantify  the  benefit  to  a 
pilot's  ability  to  maintain  correct  spatial  orientation,  that  is,  to  avoid  unrecognised 
SD,  and  recover  from  recognized  SD;  most  importantly  to  identify  when  multi-sensory 
information  may  be  of  a  benefit  to  or  hinder  pilot  spatial  orientation  in-flight. 

3.1.2  Objectives 

The  objectives  of  this  preliminary  research  programme  were  to:  - 

•  Determine  experimentally  whether,  and  by  how  much,  a  tactile  signal  can 
benefit  an  individual  carrying  out  a  high  workload  visual  task. 

•  Determine  and  quantify  the  effects  of  congruent  and  incongruent  tactile- 
visual  information  on  performance. 

•  Ascertain  if  any  beneficial  effect  is  as  a  result  of  true  multisensory 
integration  or  whether  it  is  a  simple  alerting  (attention  grabbing)  effect. 

3.2  Apparatus 

Vibrotactile  stimulation  was  delivered  with  two  EAI  Acoustics  C2  factors.  They  were 
30mm  in  diameter,  10  mm  deep,  and  weighed  17g.  Each  factor  contained  a  8mm- 
diameter  contactor.  The  factor  is  capable  of  producing  displacements  in  excess  of 
5.8mm  into  most  skin  sites. 

Two  factors  were  mounted  on  a  belt  in  such  a  way  that  that  they  rested  on  the 
subjects  oblique  muscles  (approximately  150  mm  from  the  umbilicus).  During  the 
study  there  was  no  variation  of  the  amplitude  of  the  factor  signal.  If  used,  a  factor 
was  activated  for  a  1  second  period  at  full  displacement  amplitude.  The  experimental 
study  was  carried  out  using  a  Dell™  laptop  computer,  which  managed  presentation 
of  the  visual  tasks,  controlled  the  factors  via  a  custom  controller  using  a  parallel  port 
interface  and  recorded  subject  responses. 
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Figure  3-1  A  photograph  of  the  EAI  C2  stimulus  delivery  tactor  with  a  UK  £2  coin  for  size 
comparison.  The  tactor  also  has  an  accelerometer  mounted  on  it,  for  use  in  other 
experiments  into  the  effects  of  whole  body  vibration  on  tactor  threshold  sensitivity. 


3.3  Subjects 

Eight  subjects,  five  female,  and  three  male,  (mean  age  29.6  years,  standard  deviation 
7  years)  participated  in  the  experiment.  They  were  chosen  at  random  from  staff  in 
the  Centre  for  Human  Sciences.  All  had  either  normal  or  corrected  to  normal  vision 
and  were  naive  to  the  purpose  of  the  experiment  The  experiment  was  conducted 
under  a  generic  protocol  approved  by  the  QinetiQ  Ethics  committee.  Subjects 
provided  written  consent  to  take  part  in  the  experiment 

3.4  Methods 

A  visual  task  was  developed  which  was  an  abstraction  of  the  tasks  undertaken  by  a 
pilot  and  which  contained  two  elements.  The  first  was  a  tracking  component,  which 
corresponded  to  the  pilot  maintaining  the  aircraft  in  safe  flight  or  a  secondary  target¬ 
tracking  task  for  example.  The  second  component  was  a  high  demand  cognitive  task, 
which  represents  the  many  other  tasks  a  pilot  might  undertake  in  a  high  workload 
situation. 

Subjects  were  required  to  divide  attention  between  two  visual  tasks  and  provide 
responses  to  both  simultaneously.  They  were  required  to  maximise  their 
performance  on  both  elements  of  the  task.  The  schematic  diagram  in  figure  3-2 
below  illustrates  how  the  task  appeared  to  the  subjects.  The  visual-tracking  task  was 
in  the  lower  half  of  the  screen,  and  the  alphabet  based  high-demand  cognitive  task 
was  in  the  upper  half  of  the  screen. 
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Figure  3-2.  A  diagrammatic  representation  of  the  visual  task  undertaken  by  the 
subjects.  In  this  example  the  letter  offset  is  positive  (+)  and  is  a  numerical  offset  of  3. 

Tracking  Task:  The  lower  half  of  the  display  contains  a  rectangular  box  region 
containing  a  central  green  area  and  two  separate  red  regions  on  either  side.  During 
the  experiment  a  vertical  black  line  moved  left  and  right  horizontally  in  a  semi¬ 
random  manner  (a  normally  distributed  random  number  generator  modulated  a 
deterministic  left-right  oscillation).  The  line  could  move  within  the  whole  extent  of 
the  rectangular  box  region.  The  subject  was  instructed  to  ensure  the  line  did  not 
enter  the  red  "danger  zones”  on  either  side  of  the  central  green  ("safe”)  region.  If  the 
line  did  move  into  these  zones,  the  subject  was  required  to  reset  the  line  to  the  centre 
of  the  green  zone  using  the  appropriate  reset  buttons  on  the  laptop  keyboard, 
assigned  to  the  Z  and  X  keys  respectively.  Response  time  was  a  key  measure  in  this 
experiment;  therefore  subjects  were  instructed  to  rest  the  index  and  middle  fingers 
of  the  left  hand  on  these  keys  throughout  the  experiment,  so  that  psychomotor 
influences  were  kept  to  a  minimum.  (We  stress  that  this  is  not  a  tracking  task  in  the 
traditional  sense.  Subjects  were  not  required  to  maintain  a  cursor  or  other  marker  on 
the  bar  as  it  moved.  Only  deviations  across  the  two  boundaries  and  the  time  taken  to 
cortrectthem  were  recorded). 

Alphabet  calculation  distance  task:  The  upper  half  of  the  display  contains  a  cognitive 
task.  It  was  a  five  alternative  forced  choice.  The  subject  is  initially  presented  with  a 
letter  pair  at  the  centre-top  of  the  display.  To  the  right  of  the  letter  pair  was  a 
number  with  a  sign  (either  positive  or  negative).  Below  the  letter  pair  were  five  pairs 
of  letters.  The  subject  was  required  to  compute  the  alphabetic  offset  from  the  test 
letters  by  the  amount  indicated  by  the  sign  and  number  on  the  right  and  then  choose 
from  the  five  options  presented  under  the  test  letter  pair.  Thus,  in  figure  3-2  above, 
the  correct  response  is  the  letter  set  EM,  which  is  +3  letters  away  from  BJ.  The 
subjects  indicated  their  choice  by  clicking  the  mouse  on  one  of  the  five  options.  The 
subject  had  a  limited  time  to  perform  this  task  (15  seconds).  As  soon  as  the  subject 
chose  a  letter  pair  or  at  the  end  of  the  time  period  a  new  letter  pair  and  a  different 
sign  and  offset  was  presented.  The  letter  task  thus  ran  continuously  and 
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independently  from  the  tracking  task.  The  subject  choice  of  letters  was  recorded,  or 
if  the  time  expired  the  event  was  recorded  as  a  “miss”. 

The  baseline  experiment  was  the  situation  where  the  subject  carried  out  the  two 
tasks  explained  above  with  no  other  input.  Two  additional  conditions  were  used  to 
investigate  to  what  extent  vibrotactile  information  could  be  used  to  enhance 
performance  or  reduce  error  rate  in  the  tracking  task  during  this  high-workload 
situation.  Thus  three  experimental  conditions  were  used: 


Condition  0:  (None):  The  tactors  were  present  but  were  not  used  in  this  condition. 
The  subject  needed  to  keep  looking  at  the  bar  to  see  if  it  has  gone  into  the  red  area. 


Condition  1:  (Congruent):  Whenever  the  track  line  reached  the  red  “danger”  zone  on 
the  relevant  side  the  tactor  on  the  corresponding  side  of  the  subject’s  body  was 
activated.  Thus,  if  the  line  reached  the  right  side  of  the  green  zone  the  right  tactor 
fired  and  vice  versa. 


Condition  2:  (Incongruent):  Whenever  the  track  line  reached  the  red  “danger”  zone  on 
the  relevant  side  the  tactor  on  the  opposite  side  of  the  subject’s  body  is  activated. 
Thus,  if  the  line  reached  the  right  side  in  this  case  the  left  tactor  fired  and  vice  versa. 

Subjects  were  given  one  practice  session.  This  is  equivalent  to  condition  0  above.  (In 
the  present  experiment,  we  were  interested  in  whether  subjects  could  make 
immediate  use  of  the  additional  information  from  the  tactors,  if  it  was  present; 
therefore  no  extended  training  was  provided.  In  a  real  operational  system,  possibly 
containing  many  tactors  it  is  likely  that  there  will  be  learning  effects  and  the  need  for 
training.  Such  factors  are  beyond  the  scope  of  the  present  work).  The  main 
experiment  consisted  of  the  three  conditions  detailed  above  presented  in  a  random 
order. 

The  following  data  were  collected  in  the  alphabet  task: 

•  Stimuli:  This  is  the  total  number  of  alphabet  stimuli  presentations  in  each 
condition.  This  number  is  larger  if  the  subject  achieves  a  higher  work 
rate. 

•  Correct:  The  number  of  correct  alphabet  decisions  in  each  condition. 

•  Wrong:  The  number  of  incorrect  alphabet  decisions  in  each  condition. 

•  Missed:  The  number  of  alphabet  stimuli  presented  on  which  a  decision 
was  not  made  (i.  e.  it  task  timed  out). 

•  Reaction  time:  recorded  in  seconds  for  each  stimulus  presentation. 


The  following  data  were  collected  in  the  tracking  task: 

•  Stimuli:  This  is  the  total  number  of  bar  tracking  presentations  in  each 
condition.  This  number  is  larger  if  the  subject  achieves  a  higher  work  rate 
as  measured  by  faster  resets. 

•  Correct:  The  number  of  correct  decisions  (appropriate  left  or  right  button 
pressed). 
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•  Wrong:  The  number  of  incorrect  decisions  in  each  condition 
(inappropriate  left  or  right  button  pressed). 

•  Missed:  The  number  of  tracking  errors  that  occurred  in  which  a  decision 
was  not  made. 

•  Reaction  time:  time  in  seconds  to  respond  to  the  tracking  error. 


Results 

The  results  of  the  experiment  are  summarised  in  the  three  graphs  below.  Figure  3-3 
shows  the  mean  results  for  all  eight  subjects  across  the  data  collection  categories  for 
the  alphabet  distance  task.  Figure  3-4  shows  the  mean  results  for  all  eight  subjects 
across  the  data  collection  categories  for  the  tracking  task.  Figure  3-5  shows  the 
mean  reaction  time  for  correct  responses  to  the  tracking  task  averaged  across  all 
subjects. 

Results:  Alphabet  Distance  Task 

Figure  3-3  appears  to  show  very  little  difference  between  the  three  conditions  for  any 
of  the  classification  categories.  The  number  of  correct  responses  to  the  task  was 
slightly  lower  in  the  congruent  tactor  condition.  A  repeated  measures  ANOVA  was 
performed  on  the  number  of  stimuli  and  correct  responses  respectively,  in  order  to 
determine  whether  any  statistically  significant  difference  was  evident.  The  analysis 
did  not  show  any  significant  differences.  This  implies  the  subjects  were  maintaining 
a  level  of  performance  consistently  across  the  different  tactor  conditions,  which 
would  imply  that  the  tactor  stimulation  did  not  cause  a  distraction  effect  in  either  the 
congruent  orthe  incongruent  condition. 


Alphabet  Distance  (means) 


Condition 


Figure  3-3  Alphabet  distance  task  results.  Values  are  mean  counts  averaged  across  all 
subjects. 
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3.5.2  Results:  Tracking  Task 

Figure  3-4  shows  the  results  for  the  tracking  task.  "Stimulus  count”  refers  to  the 
number  of  times  the  bar  crossed  into  the  red  zone.  Correct  refers  to  the  number  of 
times  the  appropriate  reset  button  was  pressed.  "Wrong”  refers  to  the  number  of 
times  an  inappropriate  reset  button  was  pressed,  i.e.  if  the  moving  bar  entered  the 
left  zone  but  the  subject  pressed  the  right-hand  reset  key  "Missed”  is  the  number  of 
times  the  bar  entered  the  red  zones  and  this  event  was  missed  by  the  subject. 

The  data  in  the  congruent  condition  appear  to  suggest  that  a  greater  proportion  of 
responses  were  made  correctly  than  in  the  case  when  the  tactor  input  was 
incongruent.  The  repeated  measures  ANOVA  revealed  a  significant  effect  of 
experimental  condition  on  the  number  of  stimuli  presented  [F(2,  14)  =  4.98,  p  <= 
0.05].  A  post  hoc  paired  comparisons  t-test3  revealed  that  the  number  of  tracking 
stimuli  (i.e.  the  number  of  times  the  bar  entered  the  ‘danger’  area)  in  the  congruent 
condition  was  greater  than  during  the  no  tactile  condition  [t(7)  =  -3.416,  p  <=  0.017]. 


Bar  Tracking  Task  (means) 


□  Stimuli 

□  Correct 

□  Wrong 

□  Missed 


None  Congruent  Incongruent 

Condition 


Figure  3-4  Tracking  task  results.  The  graph  shows  the  number  of  stimuli  in  each 
condition,  the  number  of  times  the  track  error  was  corrected,  the  number  of  times  the 
bar  was  not  corrected  and  the  number  missed  (i.e.  timed  out). 

A  repeated  measures  ANOVA  on  the  number  of  correct  resets  for  each  category 
revealed  a  significant  effect  of  experimental  condition  [F(2,  14)=7.686,  p<=.05],  A 
post-hoc  paired  comparison  t-test  showed  that  the  number  of  correct  resets  was 
greater  for  the  congruent  condition  compared  to  the  no  tactile  condition  [t(7)  =  - 
3.225,  p<=  0.017], 

A  repeated  measures  ANOVA  for  the  number  of  ‘wrong’  responses,  showed  a 
significant  effect  of  experimental  condition  [F(2,  14)  =  6.197,  p  <=  0.05],  A  post  hoc 
paired  comparison  t-test  revealed  that  the  there  were  more  wrong  responses  in  the 
no  tactile  condition  compared  to  the  incongruent  condition  [t(7)  =  -3.351,  p  <=  0.017], 


3  Post  hoc  t-tests  used  a  simple  Bonferroni  correction,  i.e.  (a/n)  where  a  is  the  a  priori 
significance  level. 
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The  repeated  measure  ANOVA  was  also  applied  to  the  ‘missed  responses’  category, 
when  subjects  failed  to  carry  out  a  reset  operation.  A  significant  effect  of 
experimental  condition  was  found  [F(2,14)  =  10.658,  p  <=  0.05],  The  post  hoc  paired 
comparison  t-test  revealed  that  the  were  significantly  more  missed  stimuli  in  the 
absence  of  tactile  condition  than  in  either  of  the  congruent  or  incongruent  conditions 
[t(7)  =  3.529,  p  <=  0.017]  [t(7)  =  3.265,  p  <=  0.017], 

As  well  as  analysing  the  raw  data  for  each  category,  it  was  also  decided  to  compare 
the  percentage  correct  as  a  function  of  the  number  of  stimuli  presented.  These 
percentages  were  then  put  through  repeated  measures  ANOVA,  with  a  post  hoc 
paired  comparisons  t-test  if  any  significance  was  found. 

3.5.3  Results:  Tracking  task  -  Percentage  Correct 

The  repeated  measures  ANOVA  revealed  a  significant  effect  of  condition  (F(2,  14)  = 
6.107,  p  <=  0.05).  The  post  hoc  paired  comparison  t-test  showed  that  the  significant 
difference  occurred  between  the  no  tactile  condition  and  the  incongruent  condition 
(t(7)  =  3.124,  p  <=  0.017).  The  difference  between  the  congruent  condition  and  the 
incongruent  condition  approached  significance  with  a  p  value  of  .029.  Performance 
was  certainly  more  consistent  on  the  congruent  condition  than  on  either  the 
incongruent  or  no  tactile  condition,  as  can  be  seen  from  the  low  standard  deviation 
(4.88). 


Expt  1:  Mean  Percentage  Correct :  Tracking  Task 


Condition 


Figure  3-5  Graph  showing  the  percentage  of  correct  responses  averaged  across  subjects 
for  the  tracking  task.  Error  bars  represent  +/- 1  standard  deviation  of  the  mean. 


3.5.4  Results:  Alphabet  Distance  Task  -  Percentage  Correct 

The  repeated  measures  ANOVA  revealed  no  significant  effect  of  condition.  There 
were  no  significant  differences  between  the  percentages  correct  for  any  of  the 
conditions.  In  terms  of  percentage  correct,  subjects  responded  equally  well  across  all 
conditions,  although  there  was  a  wide  degree  of  variability  amongst  different 
subjects’  performance. 
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Expt  1:  Percentage  Correct :  Alphabet  Distance 


0  1  2 

Condition 


Figure  3-6  Graph  showing  the  percentage  of  correct  responses  on  the  Alphabet  distance 
task.  Error  bars  represent  +/- 1  standard  deviation  of  the  mean. 


3.5.5  Results:  Reaction  Times 

A  graph  of  the  reaction  times  for  correct  responses  to  this  task  can  be  seen  below 
(figure  3-7).  Clearly  responses  are  much  faster  for  both  of  the  conditions  involving 
the  tactors.  The  same  procedure  was  applied  to  the  reaction  times  for  correct 
responses,  where  there  was  a  significant  effect  of  condition  [F(1.08,  7.56)  =  13.411,  p 
<=  0.05  Greenhouse-Geisser],  Post  hoc  testing  revealed  greater  reaction  times  for  the 
no  tactile  condition  compared  to  the  congruent  condition  [t(7)  =  4.029,  p<=  0.17]  and 
greater  reaction  times  for  the  incongruent  condition  compared  to  the  congruent 
condition  [t(7)  =  -4.909, p  <=  0.017], 


Tracking  Task  Reaction  Times  (means) 


(/> 

o 

a> 

a> 

£ 


c 

o 

Vi 

O 

ns 

O' 


Condition 


Figure  3-7  Reaction  times  for  the  tracking  task.  Error  bars  indicate  +/-  one  standard 
deviation  of  the  mean. 
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Discussion  of  Experiment  One 

The  aim  of  this  first  experiment  was  to  ascertain  whether  providing  a  somatosensory 
stimulus  cue  to  a  background-tracking  task  would  improve  performance  of  the  task 
under  a  high  cognitive  workload  situation.  In  addition  the  experiment  investigated 
whether  the  lateralisation  of  the  cue  influenced  performance. 

It  is  interesting  to  note  that  there  were  no  significant  differences  in  performance  on 
the  alphabet  distance  task  between  the  different  conditions,  although  the  visual 
tracking  task  showed  significant  variation.  Subjects  responded  significantly  more 
quickly  to  the  stimulus  in  both  the  active  factor  conditions.  As  expected,  the 
vibrotactile  interface  proved  to  be  an  aid  in  speed  of  response.  Furthermore,  the 
response  time  was  significantly  faster  in  condition  one,  the  congruent  condition,  than 
in  the  incongruent  condition  two.  It  can  also  be  observed  from  the  low  standard 
deviations  in  these  conditions  that  subjects  were  consistently  faster  at  responding,  as 
opposed  to  condition  0  when  there  was  no  vibrotactile  stimulation,  where  response 
times  were  higher  and  more  variable. 


The  faster  response  times  in  the  vibrotactile  conditions  meant  that  more  repetitions 
could  be  presented;  hence  there  were  significantly  more  stimuli  presentations  in  the 
congruent  condition  compared  with  the  no  factor  control  condition.  The  number  of 
stimuli  also  went  up  in  the  incongruent  condition,  although  not  significantly.  There 
were  significantly  more  correct  responses  in  the  congruent  condition  compared  with 
the  control  condition,  but  this  significant  difference  was  not  present  between  the  no 
factor  control  and  the  incongruent  condition.  The  larger  number  of  wrong  responses 
in  the  incongruent  condition  potentially  indicates  that  the  factor  cue  proved  to  be 
strong  enough  that  it  could  not  be  ignored  in  high  workload  situations;  in  some  cases 
subjects  appeared  to  instinctively  use  the  factor  cue  even  though  it  contradicted  the 
visual  information.  This  effect  suggests  that  subjects  were  using  the  lateral  nature  of 
the  factor  stimulus  to  automate  their  decisions  on  the  tracking  task. 

In  summary,  this  experiment  suggests  that: 


•  Vibrotactile  interfaces  aided  subjects  to  make  faster  responses  to  the 
visual  tracking  task. 

•  Responses  in  the  congruent  condition  were  more  likely  to  be  correct  than 
in  the  incongruent  condition. 

•  Optimised  factor  usage  enables  visual  tracking  task  performance  to 
improve  without  any  degradation  in  alphabet  distance  performance. 

These  results  are  easier  to  understand  in  a  higher-level  cognitive  framework, 
than  in  a  multisensory  integration  one.  The  factors  are  removing  from  the 
visual  domain  a  proportion  of  the  work  the  subject  must  undertake.  Because 
the  somatosensory  input  can  process  and  drive  the  required  response  without 
visual  intervention  there  is  a  greater  ability  to  concentrate  on  the  visual 
alphabet  task.  This  is  probably  not  surprising  since  only  two  factors  were  used 
and  these  mapped  clearly  onto  two  motor  responses.  Patterns  of  behaviour 
may  be  rather  different  if  more  complex  patterns  of  factor  stimulation  are  used 
and  the  task  is  not  so  clearly  lateralised. 
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4  Experiment  Two 

4.1  Introduction 

In  the  first  experiment  it  was  found  that  the  congruent  factors  were  able  to  improve 
performance  on  the  tracking  task.  However  it  was  not  clear  how  much  of  this 
improvement  in  performance  was  due  to  the  attention  diverting  component 
characteristic  of  vibrotactile  stimulation,  and  how  much  this  was  due  to  actual 
decisions  being  taken  purely  on  the  vibrotactile  information  source.  An  experiment 
was  therefore  designed  to  disambiguate  these  possibilities. 

4.2  Objectives 

In  the  second  experiment  an  alternative  condition  was  introduced.  In  this  condition, 
both  factors  activated  together  when  the  tracking  bar  entered  the  red  area  at  either 
side.  This  condition  alerted  the  subject  to  the  tracking  bar  status,  but  did  not  convey 
any  direct  information  about  the  side  that  needed  to  be  reset.  Performance  in  this 
condition  can  then  be  compared  with  performance  in  the  congruent  condition  in 
which  the  subject  was  alerted  to  the  tracking  bar’s  status  and  the  side  that  needed  to 
be  reset.  The  effect  of  the  information  component  of  the  vibrotactile  stimulus  on 
performance  on  the  bar  task  could  then  be  assessed. 

4.3  Apparatus 

The  apparatus  was  identical  to  that  used  in  experiment  one. 

4.4  Subjects 

Twelve  subjects  took  part  in  the  experiment.  They  were  chosen  at  random  from  staff 
in  the  Centre  for  Human  Sciences.  There  were  seven  female  and  five  male  subjects 
(mean  age  26.1  years,  standard  deviation  5.5  years).  All  had  either  normal  or 
corrected  to  normal  vision  and  were  naive  to  the  purpose  of  the  experiment.  The 
experiment  was  conducted  under  a  generic  protocol  approved  by  the  QinetiQ  Ethics 
committee.  Subjects  provided  written  consent  to  partake  in  the  experiment. 

4.5  Methods 

The  general  method  was  the  same  as  that  used  in  experiment  one.  The  conditions 
were: 

Condition  0:  (None):  The  factors  were  not  used  in  this  condition.  The  subject  must 
keep  looking  at  the  bar  to  see  if  it  had  gone  into  the  red  area.  This  condition  was  used 
as  a  baseline  as  in  the  first  experiment. 

Condition  1:  (Congruent):  Whenever  the  track  line  met  the  red  "danger”  zone  on  the 
relevant  side  the  factor  on  the  corresponding  side  of  the  subject’s  body  was  activated 
as  in  experiment  one. 
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Condition  2:  (Dual):  Whenever  the  track  line  met  the  red  "danger”  zone  on  either  side, 
both  tactors  were  activated  together,  thereby  alerting  the  subject  to  the  state  of  the 
tracking  bar,  but  not  to  which  side  the  bar  was  on. 

Results 

The  results  of  the  experiment  are  detailed  in  the  three  graphs  below.  Figure  3-8 
shows  the  mean  results  for  all  twelve  subjects  across  the  data  collection  categories 
for  the  alphabet  distance  task.  Figure  3-9  shows  the  mean  results  for  all  twelve 
subjects  across  the  data  collection  categories  for  the  bar  task.  Figure  3-10  shows  the 
mean  reaction  times  for  correct  responses  to  the  bar  task. 

Results:  Alphabet  Distance  Task 


Alphabet  Distance  (means)  -  Experiment  2 
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Figure  3-8  Results  from  alphabet  distance  task.  Values  plotted  are  mean  counts 
averaged  across  subjects. 

The  results  for  the  alphabet-tracking  task  are  shown  in  figure  3-8.  The  legend  is  the 
same  as  in  figure  3-3.  The  results  show  very  little  difference  between  the  three 
conditions  for  any  of  the  classification  categories.  A  repeated  measures  ANOVA  was 
performed  on  the  data.  This  did  not  reveal  any  significant  differences.  The  lack  of  any 
significant  difference  between  any  of  the  conditions  indicates  that  the  subjects 
performed  equally  well  on  the  alphabet  distance  task  for  all  conditions  as  in  the  first 
experiment. 
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4.6.2  Results:  Tracking  Task 


Expt  2:  Tracking  Task  Means 


Figure  3-9  Results  from  visual  tracking  task  (Experiment  2). 

Figure  3-9  shows  the  mean  scores  for  all  subjects  across  the  three  of  conditions.  A 
repeated  measures  ANOVA  was  used  to  test  for  any  differences  between  each 
condition. 

There  was  a  significant  effect  of  experimental  condition  on  the  number  of  stimuli 
presented  [F(2,  20)  =  5.175,  p  <=  0.05],  Post  hoc  paired  comparisons  t-tests4  revealed 
that  the  significant  difference  occurred  between  the  no  tactor  control  condition  and 
the  congruent  tactor  condition  [t(10)  =  -2.849,  p  <=  0.017],  There  was  no  significant 
difference  between  the  congruent  condition  and  the  dual  tactor  condition  however, 
or  between  the  no  tactor  condition  and  the  dual  tactor  condition. 

There  was  also  a  significant  effect  of  experimental  condition  on  the  number  of 
correct  responses  to  the  visual  tracking  task  [f(2,20)=7.097,  p<=.05].  Post  hoc  paired 
comparison  t-tests  showed  that  the  significant  difference  occurred  between  the 
control  no  tactor  condition  and  the  congruent  tactor  condition  [t(10)  =  -3.2  p  <= 
0.017],  The  differences  between  the  no  tactor  control  condition  and  the  dual  tactor 
condition,  and  between  the  congruent  tactor  condition  and  the  dual  tactor  condition 
were  not  significant. 

There  was  no  significant  effect  of  experimental  condition  on  the  number  of  wrong 
responses.  For  this  experiment  the  mean  number  of  wrong  responses  for  each 
condition  was  low:  1.45  for  the  no  tactor  control,  1.18  for  the  congruent  condition 
and  1.27  for  the  dual  condition.  The  only  missed  responses  occurred  during  the 
control  condition;  therefore  no  statistical  analysis  was  carried  out. 


4  Bonferoni  corrected  as  in  Experiment  one. 
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4.6.3 
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Expt2:  Percentage  Correct  means  :  Tracking 

Task 


Condition 


Figure  3-10  Graph  showing  the  percentage  of  correct  responses  on  the  tracking  task. 
Error  bars  indicate  +/-  one  standard  deviation  of  the  mean. 

The  repeated  measures  ANOVA  showed  a  significant  effect  of  condition  on  the 
percentage  of  correct  responses  (F(2,20)  =  4.112,  p  <=  0.05).  However,  a  post-hoc 
paired  comparison  failed  to  show  any  significance.  It  is  suggested  that  the  significant 
difference  identified  by  the  ANOVA  would  have  occurred  between  the  no  tactile 
condition  and  the  congruent  condition  (p  =  0.046),  however  the  paired  comparisons  t- 
tests  lacked  the  required  discrimination. 

Results:  Alphabet  Distance  Task 

As  in  experiment  1,  there  was  no  significant  effect  of  condition  on  the  percentage  of 
correct  responses  on  the  alphabet  distance  task. 

Results:  Reaction  Times 

The  mean  reaction  times  for  each  condition  can  be  seen  in  figure  3-12  below.  A 
repeated  measures  ANOVA  showed  a  significant  effect  of  experimental  condition 
[F(2,20)  =  8.22,  p  <=  0.001],  Post-hoc  paired  comparisons  revealed  significant 
differences  between  the  control  condition  and  the  congruent  condition  [t(10)  =  3.446, 
p  <=  0.017]  and  between  the  control  condition  and  the  dual  condition  [t(10)  =  4.012, 
p  <=  0.017].  There  was  no  significant  difference  between  reaction  times  for  the 
congruent  condition  and  the  dual  condition. 
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Expt2:  Percentage  Correct  means  :  Alphabet 
Distance 


Condition 


Figure  3-11  Graph  showing  the  percentage  of  correct  responses  on  the  alphabet 
distance  task.  Error  bars  indicate  +/-  one  standard  deviation  of  the  mean. 


Expt2:  Tracking  Task  Reaction  Times  (means) 


None  Congruent  Dual 


Condition 


Figure  3-12  Tracking  task  reaction  times  for  Experiment  2.  Error  bars  indicate  plus  and 
minus  one  standard  deviation  from  the  mean. 

Discussion  of  Experiment  Two 

Qualitatively  the  results  from  the  second  experiment  resemble  the  results  from  the 
first  experiment.  In  particular,  performance  was  maintained  in  the  alphabet  task 
under  the  different  conditions  and  differences  were  observed  in  the  reaction  times, 
both  in  terms  of  absolute  value  and  variability.  This  similarity  is  broadly  supported  by 
the  statistical  analysis. 

The  purpose  of  the  second  experiment  was  to  disambiguate  the  attention  effects  of 
the  tactors  from  the  integration  of  this  signal  with  the  visual  information.  The 
similarity  of  results,  i.e.  that  similar  effects  were  found  with  a  dual  tactor  stimulus  as 
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with  the  previous  experiment  provides  strong  support  for  the  attention  "grabbing” 
hypothesis  rather  than  any  lateral  cuing  by  the  tactor  system. 
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Conclusions  and  Recommendations 

Conclusions 

To  reiterate,  the  objectives  of  this  preliminary  research  programme  were  to:  - 


•  Determine  experimentally  whether,  and  by  how  much,  a  tactile  signal  can 
benefit  an  individual  carrying  out  a  high  workload  visual  task. 

•  Determine  and  quantify  the  effects  of  congruent  and  incongruent  tactile- 
visual  information  on  performance. 

•  Ascertain  if  any  beneficial  effect  is  as  a  result  of  true  multisensory 
integration  or  whether  it  is  a  simple  alerting  (attention  grabbing)  effect. 

We  have  used  a  simulation  of  a  pilot  activity  in  order  to  investigate  these  issues.  The 
results  have  demonstrated  that  presenting  a  tactile  stimulus  to  subjects  whilst 
undertaking  visual  tasks  does  not  distract  them  from  the  main  visual  tasks.  This  is 
evidenced  by  the  maintained  performance  in  the  alphabet  distance  task,  across  all  of 
the  experimental  conditions. 

A  tactile  stimulus  does,  however,  provide  a  benefit  in  terms  of  response  time  in 
reacting  to  the  tracking  task.  Response  times  with  tactor  support  were 
approximately  twice  as  fast  compared  with  no  tactor  support. 

In  addition,  there  was  a  much-reduced  variability  in  response  times  with  tactor 
support  compared  with  none. 

Small  differences  in  congruent  and  incongruent  tactor  support  were  observed  in  the 
data  from  the  first  experiment.  The  second  experiment  demonstrated  however  that 
the  main  effect  of  the  tactor  was  to  act  as  a  cue  to  the  track  being  lost,  i.  e.  as  an 
attention  grabbing  mechanism.  There  was  no  direct  benefit  of  the  particular 
lateralisation  of  the  cue. 

Overall  the  results  suggest  that  subjects  were  able  to  increase  capacity  and 
performance  on  the  tracking  task  without  any  detriment  to  the  high  demand 
cognitive  task  when  tactor  cueing  was  used. 

The  results  support  the  concept  of  using  a  tactor  system  as  a  method  of  aiding  a  pilot 
by  reducing  workload  and  improving  performance  and  reaction  times.  Tentatively, 
the  results  suggest  that  tactile  systems  may  reduce  the  risk  of  pilot  spatial 
disorientation  by  two  methods.  Firstly,  by  reducing  workload  and  improving 
performance  on  a  secondary  task  (e.g.  target  acquisition),  thus  minimising  distraction 
from  the  primary  flying  task.  Secondly,  by  providing  an  attention-grabbing 
mechanism  for  pilots  who  have  become  spatially  disorientated  and  are  unaware  of 
the  fact  (Type  I  SD).  However,  there  are  still  may  human  factors  issues  regarding  the 
integration  of  tactile  systems  for  pilots  that  require  further,  systematic  investigation. 
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5.2  Recommendations 

5.2.1  General 

The  study  has  identified  that  there  is  a  potential  quantitative  benefit  in  using  factors 
to  provide  cuing  information  in  a  simulation  of  a  high  workload  environment.  The 
study  was  very  much  an  initial  study  and  there  is  considerable  scope  for  further 
research  work.  Some  of  the  areas  we  believe  that  are  worth  investigating  are 
discussed  in  the  following  sections. 

5.2.2  Effects  of  different  parameters 

The  study  only  investigated  very  simple  somatosensory  system  stimulation  yet 
showed  a  significant  response  time  benefit,  albeit  in  this  simplified  task.  Many 
studies  are  currently  investigating  far  more  sophisticated  systems  consisting  of  arrays 
of  factors.  In  order  to  ascertain  the  effectiveness  and  highlight  any  potential 
detrimental  effects  of  tactile  orientation/information  aids  on  pilot  spatial 
orientation,  further  work  should  build  on  the  findings  of  the  current  scoping  study  by 
investigating  the  following: 

•  Congruent  versus  incongruent  complex  patterns  of  information  in  low  and 
high  workload  situations.  E.g.  orientation  information  provided  in  tactile 
modality,  target  acquisition  in  visual  modality.  Questions  to  be  answered 
include: 

o  Is  the  tactile  orientation  information  still  attended  to  when  moving 
from  low  to  high  workload  situations?  Is  it  any  better  in  this  situation 
than  visual  information  alone? 

o  What  is  the  effect  of  time  on  task?  Can  tactile  information  still  be 
attended  to  without  effort,  in  high  workload  visual  situations  that  last 
fortens  of  minutes  especially  if  this  information  is  incongruent?  Is  the 
tactile  information  ‘tuned  out’  after  a  while?  If  this  is  the  case  does 
the  amplitude/frequency/pattern  of  the  tactile  stimuli  need  to 
change? 

In  addition  to  these  specific  questions  there  is  a  need  to  understand  the  influence  of  a 
number  of  factors  on  the  optimal  way  of  conveying  tactile  orientation  information  to 
the  pilot  including: 

•  Tactor  location 

•  Tactor  Vibration  Frequency 

•  Tactor  Amplitude 

•  Tactor  spacing 

•  Effects  of  whole  body  vibration  (note  here  that  we  have  completed  an 
initial  programme  of  work  on  this) 

•  Effects  of  auditory  inputs  -  either  noise  or  significant  signals 

5.2.3  Brain  Activity  Correlates 

A  key  factor  associated  with  the  efficacy  of  using  factors,  as  a  mechanism  for 
augmenting  the  situation  knowledge  of  pilots  is  the  ability  to  direct  the  attention  of 
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5.2.4 
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the  wearer.  Simple  threshold  measures  of  tactor  activation  provide  an  indication  of 
sensory  activation  and  perception,  however  we  have  not  as  yet  examined  the 
potential  for  cortical  gating  of  the  sensory  input.  The  key  question  is  how  easily  is 
tactor  activity  ignored?  This  question  becomes  particularly  salient  at  periods  of  high 
workload  where  sensory  gating  could  mean  that  an  important  source  of  information 
is  ignored.  To  address  this  question  directly  it  is  proposed  that  the  steady  state 
evoked  brain  response  be  examined.  Steady-state  responses  (SSRs)  are  frequency 
following  responses  evident  in  the  electrical  activity  of  the  brain.  Crucially  SSRs  are 
attention  dependant,  thus  if  two  stimuli  are  presented  at  different  frequencies,  the 
stimuli  that  is  attended  to  will  be  augmented  whilst  the  non-attended  will  be 
attenuated. 

Saccadic  Eye  Movements 

The  traditional  mode  of  response  speed  assessment  has  been  to  measure  the  time  it 
takes  to  press  a  button,  usually  by  a  finger  or  a  foot  movement.  It  is  now  relatively 
easy  to  measure  saccadic  reaction  time,  i.e.  the  time  from  the  presentation  of  a 
stimulus  to  the  beginning  of  the  eye  movement  to  a  target.  We  do  not  know 
whether  providing  a  somatosensory  stimulus  aid  modifies  eye  movements.  This  may 
be  significant  in  a  complex  environment  such  as  a  cockpit. 

Overall  Recommendation 


This  study  used  a  set  of  tasks,  which  were  a  significant  abstraction  of  the  real  tasks 
that  a  pilot  must  undertake.  Because  a  simple  design  was  used,  the  results  were  easy 
to  interpret.  It  is  clearly  important  to  determine  whether  such  benefits  still  occur  in  a 
real  task,  since  the  diversity  and  complexity  of  flying  an  aircraft  might  negate  the 
benefits  seen  here.  Moving  to  a  flight  trial  would  not  only  be  costly  but  also  means 
that  the  experimental  control  and  rigour  is  reduced. 

We  propose  therefore  that  a  useful  compromise  would  be  to  make  use  of  the 
Cognitive  Cockpit  facility  [31]  within  the  Centre  for  Human  Sciences.  This  cockpit  has 
been  developed  by  fusing  a  number  of  technologies  including  the  real  time 


COGNITIVE  COCKPIT 
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D'lot  Monitoring  and  Aiding  Test-bed 


Tasking  Interface 
Manager  (TIM) 

Determines 
Automation 
Levels  and 
Controls  Displays 


Figure  5-1  Overview  of  Cognitive  Cockpit  Environment 
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estimation  of  cognitive-affective  status  derived  from  tracking  physiological  and 
behavioural  measures,  the  implementation  of  a  decision  support  system,  and  a 
framework  for  the  implementation  of  adaptive  automation  and  task  scheduling. 
These  are  implemented  using  principles  of  cognitive  engineering  through  a  number 
of  adaptive  interfaces.  A  closed-loop  trial  was  recently  completed  (November  2003) 
during  which  the  stability  and  performance  of  the  system  was  examined  under 
different  levels  of  threat/workload  in  a  realistic  deep-strike  mission.  Using  this 
environment  would  allow  a  number  of  aspects  -  realism  of  task,  high  workload 
environment,  monitoring  of  brain  activity,  time  on  task  and  complex  pattern  of 
information  -  all  to  be  investigated  with  a  single  controlled  experimental 
environment. 
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Psychophysical  Vibrotactile  Perceptual  Threshold  Methods 

This  annexe  summarises  some  of  the  key  experiments  on  tactor  thresholds.  Reference  citations  refer  to  the  bibliography. 


Author 

Type 

Description  of  method 

Bolanowski  S  J  Jr, 
Verrillo  RT.  (1982). 
[1] 

Modified  method 
of  limits 

A  manual  attenuator  was  adjusted  by  the  experimenter  in  2dB  steps  in  a  staircase  mode  sequentially  ascending  and 
descending  in  intensity.  The  subject’s  task  was  to  report  via  a  mechanical  switch  when  he/she  detects  any  mechanical 
event  (detection  threshold).  As  the  stimulus  intensity  decreased,  a  point  was  reached  where  the  subject  could  no  longer 
detect  the  stimulus.  The  attenuator  was  then  further  advanced  2dB  beyond  this  level  and  an  ascending  series 
commenced.  During  the  ascending  part,  the  subject  reports  where  he/she  can  detect  the  stimulus.  Several  series  of 
ascending  and  descending  profiles  are  repeated  (the  number  of  which  are  dependent  on  the  stability  of  the  subject’s 
response).  The  threshold  is  taken  as  the  geometric  mean  of  the  detection  of  the  ascending  and  descending  series. 
Sinusoidal,  thenar  eminence,  2.9cm2  contactor.  Rise  time  10  msec.  Interstimulus  interval  of  1  sec. 

Bolanowski  et  al, 
1988  [2] 

Zwislocki  et  al., 
1958. 

Forced  choice 

tracking 

Subjects  judged  which  two  sequential  intervals  of  time  contained  the  test  stimulus,  by  depressing  one  of  two  switches. 
Intervals  were  separated  by  1750  Msec.  The  stimulus  intensity  was  increased  by  ldB  for  every  response  error,  and 
decreased  by  1  dB  for  every  three  correct  responses.  Hence  the  criterion  used  to  determine  threshold  was  75%  correct 
detection  over  a  period  of  3  min.  The  actual  period  of  stimulation  was  considerably  longer  than  this  since  the  stimulus 
intensity  was  always  a  supra-threshold  level  prior  to  each  test.  Nominally,  a  period  of  3-6  minutes  was  required  before 
the  subject  reached  the  approximate  threshold  level  at  which  time  the  3-min  period  of  75%  detection  was  started. 

Craig,  1972  [3] 

Modified  Method 
of  Limits 

To  gain  an  absolute  threshold,  VT  sensation  was  raised  to  a  14-,  21-,  28-  or  35-dB  sensation  level.  Subject  depressed  a 
button  to  start  the  trial,  which  was  followed  1000msec  later  by  the  first  stimulus,  followed  by  the  second  1000msec 
later.  Subject  was  asked  to  depress  the  appropriate  button  to  indicate  which  was  more  intense  than  the  other.  Feedback 
given  by  light  indicator. 
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Craig,  1972  [3] 

Campbell,  1962 

Forced  choice  for 
detection  of 

difference 
thresholds 

Four  trials  constituted  a  block.  When  subject  gave  4  correct  responses  out  of  4  trials,  the  difference  between  the  two 
signals  was  reduced  by  0.3dB  -  i.e.  the  less  intense  of  the  2  stimuli  was  increased  in  relative  intensity  by  0.3dB.  3  correct 
responses  was  left  unchanged.  When  subject  made  2  or  less  correct  responses,  the  intensity  difference  between  the 
stimuli  were  increase  by  0.3dB.  Approx.  100  trials  (25  blocks)  arrive  at  an  estimate  of  the  difference  threshold. 

Gescheider  and 

Joelson,  1983  [4] 

Bekesy  tracking 
method 

Absolute  thresholds:  Subjects  tracked  their  thresholds  for  detecting  VT  stimuli  of  25,  40,  80  and  200  Hz,  delivered  for 
durations  of  30,  50,  etc  msecs  (this  expt  investigated  temporal  summation).  Subjects  controlled  a  hand  switch  to 
maintain  stimulus  intensity  near  threshold.  The  time  interval  between  stimuli  was  1000msec.  When  the  switch  was 
depressed,  stimulus  intensity  decreased  by  1.0  dB/sec,  and  when  the  subject  released  the  switch,  it  increased  by  1.0 
dB/sec.  Increases  and  decreases  in  intensity  over  2-4  mins  were  recorded,  and  thresholds  derived  by  fitting  a  horizontal 
line  through  the  subject’s  record. 

Gescheider  and 

Joelson,  1983  [4] 

Method  of 

magnitude 

For  ascertaining  the  intensities  that  result  in  an  equal  sensation  of  magnitude,  subjects  had  to  match  the  magnitude  of 
the  target  stimulus  on  a  secondary  stimulus,  when  stimulus  duration  varied  from  30-1000  msec.  This  comprised  two 
stimuli  separated  by  1000msec  presented  every  5  seconds.  The  duration  of  the  first  stimuli  was  1000msec,  fixed  at 
intensity  at  5,  10,  20,  30,  or  40  dB  above  threshold.  The  duration  of  the  second  stimulus  was  30,  50,  100,  200,  300,  600 
and  1000  msec.  Subjects  had  to  equate  the  sensation  of  the  magnitude  by  adjusting  the  intensity  of  the  second  stimulus. 
Closed  switch  to  increased  and  opened  to  decrease  intensity.  When  Ss  satisfied,  measurement  taken. 

Frisina  and 

Gescheider,  1977  [5] 

Bekesy  tracking 
method 

Sinuosoidal  with  1  sec  on,  1  sec  off,  100msec  rise  &  decay  time.  Increases  and  decreases  in  intensity  that  resulted  in  from 
the  subjects’  threshold  tracking  were  record  as  a  function  of  time.  Subjects  tracked  their  thresholds  for  1-2  mins  for  each 
experiment  condition.  For  each  stimulus  frequency  three  thresholds  were  obtained. 

Gescheider  GA  and 
Wright  J.H.  (1968). 
[6] 

Method  of  limits 

Subjects  tracked  threshold  response  to  the  60  Hz  stimulus  (presented  to  the  fingertip)  by  depressing  and  releasing  a  foot 
operated  switch.  When  the  switch  remained  depressed,  the  intensity  of  the  stimulus  decreased  continuously  at  a  rate  of 
ldB/sec.  When  the  switch  was  released,  intensity  increased  at  the  same  rate.  25  msec  rise/decay  time. 
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Gescheider  et  al 
1992 


Two-interval 
forced  choice 
(Zwislocki  1958) 


2  x  730  msec  pedestal  (300  Hz)  separated  by  1000  msec.  Signal  300  msec  (300  Hz),  within  pedestal.  Pedestal  and  signal 
had  rise/fall  time  of  25msec.  Observer  had  to  indicate,  by  pressing  one  of  2  buttons,  in  which  time  interval  the  signal  had 
occurred.  Next  trial  did  not  begin  until  this  provided.  Signal  appeared  at  random  in  only  one  of  the  two.  Amplitude  of 
signal  increased  by  ldB  after  each  incorrect  response,  and  decreased  by  1  dB  after  every  third  correct  response. 
Continued  until  range  of  signal  intensities  for  preceding  2  mins  (about  30  trials)  was  less  than  4  dB.  The  final  threshold 
estimate  was  centre  of  this  range.  The  procedure  provides  estimate  of  intensity  at  which  the  probability  of  a  correct 
response  is  75%. 
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